Multimodal Approach for Emotion Recognition Using a Formal Computational Model

International audience— Emotions play a crucial role in human-computer interaction. They are generally expressed and perceived through multiple modalities such as speech, facial expressions, physiological signals. Indeed, the complexity of emotions makes the acquisition very difficult and makes unimodal systems (i.e., the observation of only one source of emotion) unreliable and often unfeasible in applications of high complexity. Moreover the lack of a standard in human emotions modeling hinders the sharing of affective information between applications. In this paper, we present a multimodal approach for the emotion recognition from many sources of information. This paper aims to provide a multi-modal system for emotion recognition and exchange that will facilitate inter-systems exchanges and improve the credibility of emotional interaction between users and computers. We elaborate a multimodal emotion recognition method from Physiological Data based on signal processing algorithms. Our method permits to recognize emotion composed of several aspects like simulated and masked emotions. This method uses a new multidimensional model to represent emotional states based on an algebraic representation. The experimental results show that the proposed multimodal emotion recognition method improves the recognition rates in comparison to the unimodal approach. Compared to the state of art multimodal techniques, the proposed method gives a good results with 72% of correct

Modeling, detection and annotation of emotional states using an algebraic multidimensional vector space

Notre travail s'inscrit dans le domaine de l'affective computing et plus précisément la modélisation, détection et annotation des émotions. L'objectif est d'étudier, d'identifier et de modéliser les émotions afin d'assurer l’échange entre applications multimodales. Notre contribution s'axe donc sur trois points. En premier lieu, nous présentons une nouvelle vision de la modélisation des états émotionnels basée sur un modèle générique pour la représentation et l'échange des émotions entre applications multimodales. Il s'agit d'un modèle de représentation hiérarchique composé de trois couches distinctes : la couche psychologique, la couche de calcul formel et la couche langage. Ce modèle permet la représentation d'une infinité d'émotions et la modélisation aussi bien des émotions de base comme la colère, la tristesse et la peur que les émotions complexes comme les émotions simulées et masquées. Le second point de notre contribution est axé sur une approche monomodale de reconnaissance des émotions fondée sur l'analyse des signaux physiologiques. L'algorithme de reconnaissance des émotions s'appuie à la fois sur l'application des techniques de traitement du signal, sur une classification par plus proche voisins et également sur notre modèle multidimensionnel de représentation des émotions. Notre troisième contribution porte sur une approche multimodale de reconnaissance des émotions. Cette approche de traitement des données conduit à une génération d'information de meilleure qualité et plus fiable que celle obtenue à partir d'une seule modalité. Les résultats expérimentaux montrent une amélioration significative des taux de reconnaissance des huit émotions par rapport aux résultats obtenus avec l'approche monomodale. Enfin nous avons intégré notre travail dans une application de détection de la dépression des personnes âgées dans un habitat intelligent. Nous avons utilisé les signaux physiologiques recueillis à partir de différents capteurs installés dans l'habitat pour estimer l'état affectif de la personne concernée.This study focuses on affective computing in both fields of modeling and detecting emotions. Our contributions concern three points. First, we present a generic solution of emotional data exchange between heterogeneous multi-modal applications. This proposal is based on a new algebraic representation of emotions and is composed of three distinct layers : the psychological layer, the formal computational layer and the language layer. The first layer represents the psychological theory adopted in our approach which is the Plutchik's theory. The second layer is based on a formal multidimensional model. It matches the psychological approach of the previous layer. The final layer uses XML to generate the final emotional data to be transferred through the network. In this study we demonstrate the effectiveness of our model to represent an in infinity of emotions and to model not only the basic emotions (e.g., anger, sadness, fear) but also complex emotions like simulated and masked emotions. Moreover, our proposal provides powerful mathematical tools for the analysis and the processing of these emotions and it enables the exchange of the emotional states regardless of the modalities and sensors used in the detection step. The second contribution consists on a new monomodal method of recognizing emotional states from physiological signals. The proposed method uses signal processing techniques to analyze physiological signals. It consists of two main steps : the training step and the detection step. In the First step, our algorithm extracts the features of emotion from the data to generate an emotion training data base. Then in the second step, we apply the k-nearest-neighbor classifier to assign the predefined classes to instances in the test set. The final result is defined as an eight components vector representing the felt emotion in multidimensional space. The third contribution is focused on multimodal approach for the emotion recognition that integrates information coming from different cues and modalities. It is based on our proposed formal multidimensional model. Experimental results show how the proposed approach increases the recognition rates in comparison with the unimodal approach. Finally, we integrated our study on an automatic tool for prevention and early detection of depression using physiological sensors. It consists of two main steps : the capture of physiological features and analysis of emotional information. The first step permits to detect emotions felt throughout the day. The second step consists on analyzing these emotional information to prevent depression

Modélisation, détection et annotation des états émotionnels à l'aide d'un espace vectoriel multidimensionnel

This study focuses on affective computing in both fields of modeling and detecting emotions. Our contributions concern three points. First, we present a generic solution of emotional data exchange between heterogeneous multi-modal applications. This proposal is based on a new algebraic representation of emotions and is composed of three distinct layers : the psychological layer, the formal computational layer and the language layer. The first layer represents the psychological theory adopted in our approach which is the Plutchik's theory. The second layer is based on a formal multidimensional model. It matches the psychological approach of the previous layer. The final layer uses XML to generate the final emotional data to be transferred through the network. In this study we demonstrate the effectiveness of our model to represent an in infinity of emotions and to model not only the basic emotions (e.g., anger, sadness, fear) but also complex emotions like simulated and masked emotions. Moreover, our proposal provides powerful mathematical tools for the analysis and the processing of these emotions and it enables the exchange of the emotional states regardless of the modalities and sensors used in the detection step. The second contribution consists on a new monomodal method of recognizing emotional states from physiological signals. The proposed method uses signal processing techniques to analyze physiological signals. It consists of two main steps : the training step and the detection step. In the First step, our algorithm extracts the features of emotion from the data to generate an emotion training data base. Then in the second step, we apply the k-nearest-neighbor classifier to assign the predefined classes to instances in the test set. The final result is defined as an eight components vector representing the felt emotion in multidimensional space. The third contribution is focused on multimodal approach for the emotion recognition that integrates information coming from different cues and modalities. It is based on our proposed formal multidimensional model. Experimental results show how the proposed approach increases the recognition rates in comparison with the unimodal approach. Finally, we integrated our study on an automatic tool for prevention and early detection of depression using physiological sensors. It consists of two main steps : the capture of physiological features and analysis of emotional information. The first step permits to detect emotions felt throughout the day. The second step consists on analyzing these emotional information to prevent depression.Notre travail s'inscrit dans le domaine de l'affective computing et plus précisément la modélisation, détection et annotation des émotions. L'objectif est d'étudier, d'identifier et de modéliser les émotions afin d'assurer l’échange entre applications multimodales. Notre contribution s'axe donc sur trois points. En premier lieu, nous présentons une nouvelle vision de la modélisation des états émotionnels basée sur un modèle générique pour la représentation et l'échange des émotions entre applications multimodales. Il s'agit d'un modèle de représentation hiérarchique composé de trois couches distinctes : la couche psychologique, la couche de calcul formel et la couche langage. Ce modèle permet la représentation d'une infinité d'émotions et la modélisation aussi bien des émotions de base comme la colère, la tristesse et la peur que les émotions complexes comme les émotions simulées et masquées. Le second point de notre contribution est axé sur une approche monomodale de reconnaissance des émotions fondée sur l'analyse des signaux physiologiques. L'algorithme de reconnaissance des émotions s'appuie à la fois sur l'application des techniques de traitement du signal, sur une classification par plus proche voisins et également sur notre modèle multidimensionnel de représentation des émotions. Notre troisième contribution porte sur une approche multimodale de reconnaissance des émotions. Cette approche de traitement des données conduit à une génération d'information de meilleure qualité et plus fiable que celle obtenue à partir d'une seule modalité. Les résultats expérimentaux montrent une amélioration significative des taux de reconnaissance des huit émotions par rapport aux résultats obtenus avec l'approche monomodale. Enfin nous avons intégré notre travail dans une application de détection de la dépression des personnes âgées dans un habitat intelligent. Nous avons utilisé les signaux physiologiques recueillis à partir de différents capteurs installés dans l'habitat pour estimer l'état affectif de la personne concernée

Emotion Recognition Using KNN Classification for User Modeling and Sharing of Affect States

International audienceIn this study, we propose a new method of recognizing emotional states from physiological signals. Our proposal uses signal processing techniques to analyze physiological signals. It permits to recognize not only the basic emotions (e.g., anger, sadness, fear) but also any kind of complex emotion, including simultaneous superposed or masked emotions. This method consists of two main steps: the training step and the detection step. In the First step, our algorithm extracts the features of emotion from the data to generate an emotion training data base. Then in the second step, we apply the k-nearest-neighbor classifier to assign the predefined classes to instances in the test set. The final result is defined as an eight components vector representing emotion in multidimensional space. Experiments show the efficiency of the proposed method in detecting basic emotion by giving hight recognition rate

Modélisation, détection et annotation des états émotionnels à l'aide d'un espace vectoriel multidimensionnel

Notre travail s'inscrit dans le domaine de l'affective computing et plus précisément la modélisation, détection et annotation des émotions. L'objectif est d'étudier, d'identifier et de modéliser les émotions afin d'assurer l échange entre applications multimodales. Notre contribution s'axe donc sur trois points. En premier lieu, nous présentons une nouvelle vision de la modélisation des états émotionnels basée sur un modèle générique pour la représentation et l'échange des émotions entre applications multimodales. Il s'agit d'un modèle de représentation hiérarchique composé de trois couches distinctes : la couche psychologique, la couche de calcul formel et la couche langage. Ce modèle permet la représentation d'une infinité d'émotions et la modélisation aussi bien des émotions de base comme la colère, la tristesse et la peur que les émotions complexes comme les émotions simulées et masquées. Le second point de notre contribution est axé sur une approche monomodale de reconnaissance des émotions fondée sur l'analyse des signaux physiologiques. L'algorithme de reconnaissance des émotions s'appuie à la fois sur l'application des techniques de traitement du signal, sur une classification par plus proche voisins et également sur notre modèle multidimensionnel de représentation des émotions. Notre troisième contribution porte sur une approche multimodale de reconnaissance des émotions. Cette approche de traitement des données conduit à une génération d'information de meilleure qualité et plus fiable que celle obtenue à partir d'une seule modalité. Les résultats expérimentaux montrent une amélioration significative des taux de reconnaissance des huit émotions par rapport aux résultats obtenus avec l'approche monomodale. Enfin nous avons intégré notre travail dans une application de détection de la dépression des personnes âgées dans un habitat intelligent. Nous avons utilisé les signaux physiologiques recueillis à partir de différents capteurs installés dans l'habitat pour estimer l'état affectif de la personne concernée.This study focuses on affective computing in both fields of modeling and detecting emotions. Our contributions concern three points. First, we present a generic solution of emotional data exchange between heterogeneous multi-modal applications. This proposal is based on a new algebraic representation of emotions and is composed of three distinct layers : the psychological layer, the formal computational layer and the language layer. The first layer represents the psychological theory adopted in our approach which is the Plutchik's theory. The second layer is based on a formal multidimensional model. It matches the psychological approach of the previous layer. The final layer uses XML to generate the final emotional data to be transferred through the network. In this study we demonstrate the effectiveness of our model to represent an in infinity of emotions and to model not only the basic emotions (e.g., anger, sadness, fear) but also complex emotions like simulated and masked emotions. Moreover, our proposal provides powerful mathematical tools for the analysis and the processing of these emotions and it enables the exchange of the emotional states regardless of the modalities and sensors used in the detection step. The second contribution consists on a new monomodal method of recognizing emotional states from physiological signals. The proposed method uses signal processing techniques to analyze physiological signals. It consists of two main steps : the training step and the detection step. In the First step, our algorithm extracts the features of emotion from the data to generate an emotion training data base. Then in the second step, we apply the k-nearest-neighbor classifier to assign the predefined classes to instances in the test set. The final result is defined as an eight components vector representing the felt emotion in multidimensional space. The third contribution is focused on multimodal approach for the emotion recognition that integrates information coming from different cues and modalities. It is based on our proposed formal multidimensional model. Experimental results show how the proposed approach increases the recognition rates in comparison with the unimodal approach. Finally, we integrated our study on an automatic tool for prevention and early detection of depression using physiological sensors. It consists of two main steps : the capture of physiological features and analysis of emotional information. The first step permits to detect emotions felt throughout the day. The second step consists on analyzing these emotional information to prevent depression.NICE-Bibliotheque electronique (060889901) / SudocSudocFranceF

Multimodal recognition of emotions using a formal computational model

International audienceIn this paper, we present a multimodal approach for the emotion recognition that takes into account more sources of information (physiological signals, facial expressions, speech, etc). This approach is based on an algebraic representation of emotional states using multidimensional vectors. This multidimensional model provides a powerful mathematical tools for the analysis and the processing of emotions. It permits to integrate information from different modalities (speech, facial expressions, gestures) in order to allow more reliable estimation of emotional states. Indeed, our proposal aims at efficient recognition of emotional state even when it appear to be superposed or masked

Detecting depression using multimodal approach of emotion recognition

International audienceDepression is a growing problem in our society. It causes pain and suffering not only to patients but also to those who care about them. This paper presents a multimodal emotion recognition system that is capable of preventing depression. It consists of detecting persistent negative emotions for early detection of depression. Our proposal is based on an algebraic representation of emotional states using multidimensional vectors. This algebraic model provides powerful mathematical tools for the analysis and the processing of emotions and permits the fusion of complementary information such as facial expression, voice, physiological signals, etc. Experiments results show the efficiency of the proposed method in detecting negative emotions by giving high recognition rate

Multimodal Approach for Emotion Recognition Using an Algebraic Representation of Emotional States

International audienceEmotions play a key role in human-computer interaction. They are generally expressed through several ways (e.g. facial expressions, speech, body postures and gestures, etc). In this paper, we present a multimodal approach for the emotion recognition that integrates information coming from different cues and modalities. It is based on a formal multidimensional model using an algebraic representation of emotional states. This multidimensional model provides to represent infinity of emotions and provide powerful mathematical tools for the analysis and the processing of these emotions. It permits to estimate the human emotional state through combining information from different modalities (e.g. facial expressions, speech, body postures and gestures, etc) in order to allow more reliable estimation of emotional states. Our proposal permits to recognize not only the basic emotions (e.g., anger, sadness, fear) but also different types of complex emotions like simulated and masked emotions. Experimental results show how the proposed approach increase the recognition rates in comparison with the unimodal approach

Multimodal Recognition of Emotions Using Physiological Signals with the Method of Decision-Level Fusion for Healthcare Applications

International audienceAutomatic emotion recognition enhance dramatically the development of human/machine dialogue. Indeed, it allows computers to determine the emotion felt by the user and adapt consequently its behavior. This paper presents a new method for the fusion of signals for the purpose of a multimodal recognition of eight basic emotions using physiological signals. After a learning phase where an emotion data base is constructed, we apply the recognition algorithm on each modality separately. Then, we merge all these decisions separately by applying a decision fusion approach to improve recognition rate. The experiments show that the proposed method allows high accuracy emotion recognition. Indeed we get a recognition rate of 81.69% under some conditions

Detecting and Classification of the Basic Emotions Using a Multimodal Approach for Emotions Detection

International audienc